Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Support gzipped user data #1366

Merged
merged 2 commits into from
Mar 9, 2021
Merged

Support gzipped user data #1366

merged 2 commits into from
Mar 9, 2021

Conversation

tjkirch
Copy link
Contributor

@tjkirch tjkirch commented Mar 5, 2021

Issue number:

Fixes #1320. The goal is adding an easy way to handle larger amounts of user data, now that we support adding static pods, host container user data, and other settings that could potentially be long. (This isn't the only solution or the final one, but it's an easy use case.)

Description of changes:

778760a7 early-boot-config: add module to handle maybe-gzip inputs
7a558d9c early-boot-config: allow gzip compression of user data

Note: this is based on #1365 because I needed the model fix to properly run the vmware-dev variant.

Testing done:

aws-k8s-1.17

Started an instance with aws-cli's --user-data fileb://path argument pointing to a gzipped file with >16KiB of text uncompressed, saw those settings take effect (very long motd), and the instance was healthy.

Started the same AMI with a (smaller) uncompressed user data file and was still successful/healthy.

aws-dev

Used an uncompressed local user data file OK:

$ sudo sheltie
bash-5.0# mkdir /etc/early-boot-config
bash-5.0# echo -e '[settings]\nmotd = "woo"' > /etc/early-boot-config/user-data
bash-5.0# early-boot-config
bash-5.0# settings-committer
bash-5.0# thar-be-settings --all
bash-5.0# cat /etc/motd
woo

Then I did the same on a new instance, but outside of sheltie (so gzip is available) with echo -e '[settings]\nmotd = "compressed"' | gzip | sudo tee user-data. Still saw the motd update OK.

vmware-dev

I ran vmware-dev in qemu, and tested changing the motd with a plain uncompressed TOML file in a ISO user data; worked fine.

I tested with a compressed user-data file OK, confirming that it's compressed at /media/cdrom/user-data but uncompressed OK into settings.motd and the motd file.

Then I repeated the tests with an OVF file inside the ISO. Plain text was OK, I confirmed /media/cdrom/ovf_env.xml has the plain text base64, and the API/motd shows the text. Compressed was OK, I confirmed /media/cdrom/ovf_env.xml has the compressed text base64, and the API/motd shows uncompressed text.

Terms of contribution:

By submitting this pull request, I agree that this contribution is dual-licensed under the terms of both the Apache License, version 2.0, and the MIT license.

@tjkirch tjkirch requested review from bcressey and zmrow March 5, 2021 01:07
@tjkirch
Copy link
Contributor Author

tjkirch commented Mar 8, 2021

^ This push clarifies the comment about file magic, and explicitly changes the gzip const length to 2, to represent the fact that it may differ in the future.

@tjkirch
Copy link
Contributor Author

tjkirch commented Mar 9, 2021

^ @bcressey gave the nice suggestion of a unit test that confirms giving part of the magic string works, in that it just gives back the input, it's not falsely recognized as gzip or anything. Added that in this push; test passes.

@tjkirch
Copy link
Contributor Author

tjkirch commented Mar 9, 2021

^ This push moves the main type to a struct, per @webern.

Copy link
Contributor

@zmrow zmrow left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔬

@tjkirch tjkirch merged this pull request into vmware-no-metrics Mar 9, 2021
@tjkirch tjkirch deleted the gzip-user-data branch March 9, 2021 22:12
tjkirch added a commit that referenced this pull request Mar 9, 2021
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants